Simplified Tool Call Accuracy V1 #40710
Conversation
github-actions
bot
added
the
Evaluation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a simplified mechanism for predicting and evaluating tool call accuracy by adding a new tool call predictor module and a corresponding evaluator.
- Added the predict_tools function in _tool_call_predictor.py to generate ground truth based on a prompt model.
- Exposed predict_tools via an init.py file for easier consumption.
- Included a new evaluator in _tool_accuracy_new.py that implements scoring logic for tool calls and tool results.
Reviewed Changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_calls_predictor/_tool_call_predictor.py | Implements the tool call predictor using prompt models (contains duplicate imports that could lead to ambiguity). |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_calls_predictor/init.py | Exposes the predict_tools function for the caller. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_call_accuracy_new.py | Introduces new evaluation logic for tool call accuracy including extraction and match scoring. |
Files not reviewed (1)
- sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_calls_predictor/tool_call_predictor.prompty: Language not supported
Comments suppressed due to low confidence (2)
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_calls_predictor/_tool_call_predictor.py:29
- Duplicate import of AsyncPrompty detected (also imported on line 10). Consider consolidating the imports to a single source to avoid ambiguity.
from promptflow.core import AsyncPrompty
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_call_accuracy_new.py:151
- The function 'generate_ground_truth' is referenced but not defined or imported; please add its definition or import it.
ground_truth = generate_ground_truth(query, response, tool_definitions)
|
API change check API changes are not detected in this pull request. |
|
@ghyadav Can you please share context on requirements for a new tool call accuracy evaluator ? |
|
Hi @ghyadav. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days. |
github-actions
bot
added
the
no-recent-activity
|
Hi @ghyadav. Thank you for your contribution. Since there hasn't been recent engagement, we're going to close this out. Feel free to respond with a comment containing |
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines